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Abstract 



After fifteen years, the Wisconsin Secondary Vocational Education Evaluation Project was 
phased out in 1991. With the reauthorization of the Carl Perkins Vocational Education Act at the 
federal level, the distribution of funds was changed and the Wisconsm Department of Public 
Instruction was no longer able to fund the evaluation project. A number of secondary vocational 
educators voiced concern about the loss of the evaluation project and the services it provided. 
They encouraged CVTAE staff members to develop a revised evaluation system and make it 
available to secondary schools. Many of these vocational educators also offered to help with the 
revision process. 

The purpose of this project was to identify the program evaluation needs of secondary vocational 
educators in Wisconsin. In addition, the project was designed to capture the expertise and 
evaluation experience of a number of secondary vocational educators. The project was designed 
to identify the components of the previous evaluation system that educators wanted to keep and 
to obtain information on changes that should be made. Four input sessions were conducted 
around the State of Wisconsin to gather this input. These were completed during the fall of 
1992. After the input session had been completed, the suggestions from each session were 
combined and a survey form was developed. This survey was mailed to everyone who signed up 
for one of the input sessions. Recipients of the survey were asked to place priorities on each of 
the suggestions given. 

Participants in the input sessions and survey recommended that the self-evaluation, on-site 
review process, and training for evaluators be retained. They also indicated a need for a more 
flexible evaluation process, additional emphasis on evaluating local plans, and expanding the 
evaluation to include their school-to- work transition program. 

A framework for a revised evaluation system was designed. Systems and TQM concepts were 
used to develop the framework. Emphasis was placed on evaluating the district's plan for the 
future in place of evaluating what had been done. Districts were given more flexibility in 
selecting evaluation team members and data collection techniques. 
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Conceptual Framework for Revising the Secondary Vocational Education 

Program Evaluation System 



Purpose and Objectives 

Introduction 

In 1975, the Center for Vocational, Technical and Adult Education (CVTAE) received a call 
from several LVECs who were concerned about improving their secondary vocational education 
programs through the use of effective program evaluation techniques. The Center directors met 
with these LVECs, school administrators, and representatives from other vocational education 
programs, including the technical college system. Participaiiis in the meeting identified the types 
of data that were needed to improve secondary vocational education programs. This group 
became the first advisory committee for the evaluation project. 

This input was used by CVTAE staff members to develop the initial evaluation design for 
secondary vocational education programs. This design was checked against the evaluation 
requirements in the federal vocational education legislation. In addition, a draft copy was 
reviewed by the advisory committee members. Changes were made to meet federal requirements 
and to incorporate the suggestions from the advisory committee. 

During the next fifteen years, this evaluation model and process were used to evaluate the 
secondary vocational education programs in Wisconsin that received federal funding. The 
model, procedures, and processes were continuously refined during the fifteen year period. At 
the end of each of the reauthorization cycles for the federal vocational education legislation, an 
advisory committee was convened to review the model and suggest improvements. 

With the advent of the reauthorization of the Carl Perkins Vocational Education Act in 1991, the 
Wisconsin Department of Public Instruction no longer had funds to sustain the evaluation project 
that provided technical assistance, in-seryice workshops and data processing to secondary 
schools. As a result the evaluation project at UW-Stout was phased out. 

After the phase out, the Center received requests from secondary vocational educators for 
assistance with program evaluation. They were aware that the standards and measures mandated 
in the federal vocational education legislation would have to be used. However, many of them 
concluded that they needed to have more than the standards and measures in order to improve 
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their programs. This project was designed to identify the evaluation processes and data needed 
by secondary vocational educators. 

Pur[)ose 

The purpose of this study was to identify the program evaluation needs of vocational educators at 
the secondaiy school level in Wisconsin. 

Objectives 

This project was designed to attain the following objectives. 

1. Identify the components in the Phase HI Secondary Vocational Education Program 
Evaluation Model and Process that secondary vocational, educators wanted to retain. 

2. Identify the components of the Secondary Vocational Program Evaluation System that 
vocational educators wished to delete or give a low priority. 

3. Develop a list of additional program facets related to school-to-work transition programs 
that needed to be evaluated. 

4. Create a synthesis of the input from secondary vocational educators and develop a set of 
prescriptions and an outline for a new evaluation system. 

Limitations 

This study was limited to developing a set of prescriptions and an outline for the evaluation of 
high school vocational education and school-to-work transition programs. It was recognized that 
these programs do not exist in isolation from other high school programs. Thus, some data was 
collected on the areas that overlap with other programs in the school district. However, the 
project had limited resources and the original point of concern was the evaluation of vocational 
programs. Other agencies were involved in providing evaluation systems and services for 
academic programs and the Center did not want to meddle in their missions. 

Previous Evaluation System 

The system used to evaluate secondary vocational education programs during the period from 
1976 to 1992 contained three phases: Self-Evaluation, External Evaluation, and Utilization. 
Activities in the three phases are described on the next page. 



ERIC 



2 

9 



Phase I 

Self-Evaluation. Adapting 
and using the procedures 
and questionnaires contained 
in the Vocational Evaluation 
Manual, schools conducted and 
reported a comprehensive self- 
evaluation of their total 
vocational program. These 
included vocational guidance 
and counseling, and Carl 
Perkins targeted populations. 
Five questionnaires were 
required; an additional eight 
were optional. Schools which 
completed an evaluation in 
the previous cycle addressed 
utilization of their findings 
toward program improvements 
as outlined in their Carl 
Perkins Plan. 



Phase II 

External Evaluation. 

Adapting and using 
procedures and instruments 
contained in the Vocational 
Evaluation Manual, schools 
hosted an external evaluation 
and received a written report 
containing conclusions, rec- 
ommendations, and solutions 
for program improvement. 
This report incorporated 
progress made toward imple- 
menting recommendations 
from the previous cycle, 
and solutions for program 
improvement. 



Phase III 

Utilization of 
Evaluation Findings. 

In this phase, schools 
analyzed and interpreted 
findings from the 
Self-Evaluation and 
Team Visitation 
Reports, set priorities, 
implemented changes, 
and monitored impact 
which occurs as a 
result of the evalua- 
tion. Participation 
of staff was acknow- 
ledged. Plans for the 
next evaluation cycle 
were also begun. The 
viorities set in the 
utilization Phase were 
used in developing 
and/or updating the 
Carl Perkins Plans 
and program/ 
departmental plans. 



The total vocational program was evaluated including special needs, guidance and counseling 
and the management components. Wisconsin DPI recognized six vocational program areas: 
agriculture education, business education, health occupations, family and consumer education, 
marketing education and technology education. Staff members were not evaluated. Statements 
in the Team Report were phrased to address program concerns. 



Information from the evaluation was useful for decision-making at three levels. Figure 1 
identifies die outcomes for local, state and federal levels. The Center for Vocational, Technical, 
and Adult Education (CVTAE) at UW-Stout processed the evaluation data for each school and 
printed LEA summaiies. At the end of each year, the Center combined the LEA data sets and 
developed a composite report. The composite summaries presented the information needed for 
the statewide planning and the Carl Perkins Vocational Education Act. 
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FIGURE 1 
Three Major Prioritized Outcomes 



1 

LOCAL 
L,bi\biLj 


To promote and assist in the evaluation of LEA* vocational 
programs for the purpose of program improvement at the 
local level. 


2 

STATE 

I 1?\71?T 

JLUrVCiL/ 


To provide valid representative data on LEA* vocational 
programs for regional and statew^ide planners of vocational 
education and to satisfy state requirements of program 
evaluation. 


3 

NATIONAL 
LEVEL 


To help ensure accountability of federal funds allotted to 
LEA* vocational programs and to satisfy federal 
requirements of program evaluation. 



LEA* Local Education Agency (K-12, or as appropriate) 



Need for a New System 

The last reauthorization of the Carl Perkins Act changed the funding formula for the LEA and 
State level activities. More funds were sent to the LEAs and there were fewer dollars at the State 
level. As a result, the evaluation project was not continued after 1992. 

A number of secondary vocational educators indicated a need for a new evaluation system and a 
desire to have input in the design process. Their requests were based on a perceived need to use 
evaluation data to improve their programs. CVTAE wanted to utilize the knowledge they had 
developed through their experiences with the original evaluation system to develop a revised 
model. 

Research Procedures 

Introduction 

As noted previously, the purpose of this project was to gain input from vocational educators and 
administrators concerning the types of evaluation data they need in order to improve vocational 
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education programs at the secondary level This project focused on collecting input from 
educators, analyzing it, and developing the framework for the evaluation system. 



Study Design 

CVTAE staff members conducted a series of four input sessions throughout the State of 
Wisconsin to obtain information from LVECs, vocational educators and high school 
administrators. Each meeting had the same format The first portion of the session used the 
brainstorming technique to generate a list of ideas. These ideas were prioritized in the last part of 
the session. 

After the four sessions had been completed. Center staff members reviewed all of the lists and 
the priorities established by &^ focus groups. This information was synthesized and used to 
develop a survey. This survey was mailed to everyone who signed up for an input session. The 
results from this survey were used to establish the final priorities. 

The results from the four sessions and the prescriptions were discussed with an advisory 
committee comprised of secondary vocational educators. Their input was used to refine the 
prescriptions and to develop an outiine of the evaluation design. 

Input Session Results 

The four input sessions for LVECs and vocational educators were scheduled in November and 
December. The sessions were held in Menomonie, Madison, Appleton, and Wausau. The 
Center for Vocational, Technical and Adult Education sent a letter to all LVECs in Wisconsin 
announcing the four sessions and inviting them to attend the session of their choice. They weie 
also encouraged to bring other vocational educators. A total of 34 educators participated in these 
sessions. 

The structure of the input sessions was developed with the assistance of the WAS VA Board. 
During the first portion of the session, the participants responded to three questions: 

• What do you want to keep from the previous evaluation system? 

• What do you want to change? 

• What additional evaluation needs do you have? 
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A brainstorming format was used to obtain this inpui. 

After the recommendations had been listed for each question, the lists were edited and 
recommendations were combined when there was significant overlap. The next step was to 
prioritize the recommendations for each question. Participants were asked to identify their top 
five priorities. The nwst important recommendation was given five points, the next most 
important four points, and the fifth priority received one point. A copy of the results from each 
session is reproduced in Appendix A. 

Table 1 presents a summary of the recommendations and priorities given at the four input 
sessions. The recommendations are divided into two sections: (1) components that should be 
retained from the previous evaluation system, and (2) components that should be changed or 
added. The information given to the right of the recommendations indicates the input session, 
priority, and where the recommendation can be found in the individual summaries. 

Recommendations in the first part of Table 1 were concerned with components that should be 
retained and come from the responses to the first question discussed at the input sessions, "What 
components of the evaluation system should we keep?" The numbers in the columns to the right 
indicate the priorities given by the participants. Stars indicate that the recommendation was 
made but the group did not prioritize the total set of recommendations. A dash (-) denotes that 
the group did not make the recommendation. 

Recommendations in part 2 came from the responses to the last two questions discussed at the 
input sessions, "What should we keep?" and "What additional evaluation needs do you have?" 

Some of the major themes in the input were: 

♦ Retain the evaluation process, especially the self-evaluation and on-site review process. 

♦ Provide more flexibility for local school districts to tailor the evaluation design lo their 
needs. 

♦ Place more emphasis on evaluating local plans and goals to determine if tlie program is 
moving in the right direction. 

♦ Include a greater variety of data collection techniques. 

♦ Allow more flexibility in selecting the on-site evaluation team, both in terms of a wider 
variety of constituencies and the use of consultants. 

♦ Expand the evaluation system to include Tech Prep, Education for Employment, and 
School-to- Work Transition Programs. 



TABLE 1 

Summary of Suggestions and Priorities From the Input Sessions 



Recommendations 



SESSION 



MEN MAD APP WAIT 



RETAIN: 

a. Self-evaluation. 

b. On-site evaluation. 

c. Graduate follow-up. 

d. Structured system for the evaluation process. 

e. Include special needs and guidance programs in 
the evaluation. 

f. Training for team leaders and team members. 

g. Key questions. 

h. Data processing services. 

i. Stout coordinator. 

j. Advisory committee to provide input on the 
evaluation system. 



2 . 

3 

5 

1 

4 

l(Sub) 
l(Sub) 



* 
* 
* 
* 



* 
* 



3 

2&4 



4 
5 
1 



* 
* 
* 



CHANGES TO MAKE: 

k. Provide more flexibility in the evaluation system. III-4 

1. Orient the evaluation to program goals and where III-4 
program should be headed. 

m. Broaden the composition of the evaluation team. III-2 

n. Allow flexibility in selecting team members. II-6 
(Peers vs. consultants, etc.) 

o. Evaluate Tech Prep, Education for Employment III- 1 
and School-to- Work Transition Programs. 

p. Assess integration of subject matter areas. III-l 

q. Make the evaluation process more efficient. (Includes II-4 
coordination with SEC, VEERS, Measures & Standards.) 

r. Expand the data collection techniques that can be 111-5 
used. (Focus groups, student performance assessment, 
authentic assessment, etc.) 

s. Improve data collection forms and processes. II-4 

II-5 

t. Incorporate new educational developments. (Outcome- III-5 

based education, business-education linbiges.) Ill- 1 



III-4 n-4 
II-5 



III-5 



II-4 



II- 7 

III- l 

II- 6 

III- l 

II-l 

II- 2 

III- 5 

II-l 
11-10 

II-l 

II-8 



II-2 
II-2 



III-l 
III-3 

II- l 

III- 2 
III-7 

II-7 
II-8 



II-D 



III-l III- A 



II-E 



II-A 



II-C 

II- E 

III- C 



Results from the input sessions were used to develop a survey instrument. This was mailed to 
everyone who signed up for an input session. 



The survey results from forty-seven vocational educators are given in Table 2. Elements from 
the first evaluation system that should be retained are given in the first section. The self- 
evaluation and the on-site evaluation were identified as very important. Also, the participants 
thought that graduate follow-up, training for team members, data processing services and the 
inclusion of special needs and guidance programs were important to very important and should 
be retained. 

Recommended changes in the evaluation system are given in section two. Evaluating the total 
school-to-work transition program, integration of subject matter areas, and efficiency received 
"very important" ratings. More flexibility in the evaluation process and selection of the 
evaluation team received "important" ratings. 

TABLE 2 

Vocational Education Program Evaluation 
System Survey Results 



SUGGESTIONS Median 

RETAIN: 

1. Self-evaluation 4.7* 

2. On-site evaluation 4.5 

3. Graduate follow-up 4.4 

4. Structured system for the evaluation 
process 4.2 

5. Include special needs and guidance 
programs in the evaluation 4.4 

6. Training for team leaders and team 
members 4.4 

7. Key questions 4.1 

8. Data processing services 4.4 

9. Stout coordinator 4.1 

10. Advisory committee to provide input 

on the evaluation system 4.2 

CHANGES TO MAKE: 

1 1 . Provide more flexibility in the 

evaluation system 4.2 



Results 



TOR 



% I&VI 



1.0 
1.2 
1.3 

1.4 

1.4 

1.3 
1.6 
1.5 
1.6 

1.6 



1.3 



96@ 

85 

79 

72 

77 

79 
68 
73 
68 

66 



74 
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Results 



SlICwGESTTONS Median IQR % I&VI 

1 2. Orient the evaluation to program goals 
and where the program should be 

headed 4,4 LI 87 

13. Broaden the composition of the 

evaluation team 3.4 1.9 47 

14. Allow flexibility in selecting team 

members (Peers, consultants, etc) 3.9 1.5 64 

15. Evaluate Tech Prep, Education for 
Employment and School-to-Work 

Transition Programs 4.8 LO 87 

16. Assess integration of subject matter 

areas 4.7 1.1 88 

17. Make the evaluation process more 
efficient. (Includes coordination with 

SEC, VEERS, Measures & Standards). . 4.7 1 .2 83 

1 8. Expand the data collection techniques 
that can be used. (Focus groups, 
student performance assessment, 

authentic assessment, etc.) 4.2 1.3 76 

19. Improve data collection forms and 

processes 4.5 13 79 

20. Incorporate new educational 
developments. (Outcome-based 
education, business-education 

linkages, etc.) 4.5 1-5 74 

N=47 

# KEY: Median and IQR {Interquartile Range} are based on the following response scale: 
l=NI=Not Important 4=I=Important 
2=SI=Slightly Important 5=VI=Very Important 

3=MI=Moderately Important 

@ Percent I&VI=Sum of the percentage of respondents that selected the Important and Very 
Important responses. 
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Introducrion 

The purpose of the evaluation framework is to identify the major concepts or components in the 
evaluation system and identify how these elements interact. The framework will provide 
direction for the detailed design and developmental work that will follow. Specific design and 
development work were not a part of this project. Thus, this section presents a broad ouUine of 
the evaluation system. 

The following sections will identify the concepts that provide the overall structure for the 
evaluation system and specific elements within the evaluation system. 

Evaluation Framework 

Two major concepts were used to guide the design of the evaluation system. These were the 
systems approach and total quality management. 

Application of the systems approach provides a very useful perspective for the design of the 
evaluation system. The systems perspe 3tive encourages a comprehensive view of the factors 
involved in a program and how they interact with each other. A system is comprised of two or 
more elements that interact to accomplish a specific purpose or set of purposes. The output from 
a system must match these purposes. Inputs are processed to attain desired outputs. In addition, 
every system operates within an environment that has an impact on the inputs, processes and 
outputs. The systems approach is depicted in Figure 2. 

A vocational program can be viewed as a system. It has a defined set of purposes which include 
preparation of students for entry into a job or advanced educational programs. The inputs into 
the system would include students with certain abilities and interests. Various instructional 
processes and learning experiences are used to change these input characteristics to competencies 
and skills that are needed on the job. The environment for this system would include the school 
climate, students' peers, and quality of the area economy. 

Systems design is carried out to create the best combination of inputs, processes, and outputs 
within a given environment. These components are selected and/or modified to create a system 
that will achieve its purposes in the most effective and efficient manner. Another design 
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principle involves the interrelationship of systems. What happens in one system often times has 
a significant effect on another system. For example, if the academic requirements needed for 
graduation are significantly changed, this also has an impact on vocational programs. Figure 3 
depicts a common relationship between academic and vocational education systems. There is a 
separation between the two and the lack of a two-way interaction between the vocational and 
academic programs. 

E):perience has indicated that the relationship between academic and vocational programs 
illustrated in Figure 3 is not an effective design. Vocational education is not a mainstream 
component of the educational program in this system. Many students who could benefit from 
various aspects of vocational education arc discouraged or blocked from access to them. 
Students who are taking vocational education often do not have access to appropriate academic 
competencies. 

A more desirable system is described in Figure 4. Learning experiences that develop academic, 
career decision-making, vocational, and community participation competencies are integrated in 
one system. They are interrelated and designed to prepare people for the various roles they will 
encounter as adults. This system includes a feedback loop from graduates. This feedback 
provides information on the adequacy and relevancy of the competencies developed in the 
educational program. When used on a systematic basis, this system will be self-correcting and 
continuously improving. 

The second set of concepts used to develop the evaluation framework comes from total quality 
management (TQM). Quality is commonly defined as "meeting customer expectations." Some 
organizations are changing this to "exceeding customer expectations." Some educational 
organizations are defining quality as "meeting client expectations." In either case quality is 
dependent upon identifying the relevant customers and their expectations. 

Total quality management (TQM) is based on the principle that quality is everyone's 
responsibility. In a school setting this would include administrators, teachers, counselors, 
support staff, parents, and students. The elements of TQM are presented in Figure 5. Each 
person in a system must understand how he or she contributes to the quality of the output from 
that system. Processes must be designed to meet customer expectations. Processes are those 
activities that modify and change inputs to produce a desired product or service. In education 
these processes focus on the learning activities and experiences of students. They also involve 
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FIGURE 2: SYSTEMS APPROACH 
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FIGURE 3: Academic and Vocational Systems 
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the activities of teachers, administrators, counselors and support staff that generate these 
educational experiences. 

TQM includes a continuous improvement systenL Various strategies are available for this. One 
approach commonly used is PDCA. This acronym represents plan, do, check and act. The 
planning stage involves using available information to plan or develop an improvement in the 
system. This plan is tried out in the doing stage. In the next stage, check, data is collected and 
the impacts of the change are evaluated. Based on this evaluation a decision is made on putting 
the new program or approach in place and using it on a regular basis (Act). The basic concept 
involved in continuous improvement is to use the PDCA cycle as a part of ongoing operations to 
refine the system, Tlie thrust is for incremental change over a period of time. 

Teamwork is an interval part of TQM. TQM encourages the interchange of ideas and joint 
action. It recognizes that systems are more productive when people work together. 

TQM also recognizes that people create change. It encourages staff development activities and 
the professional growth of individuals who work in the system. 

Based on the application of TQM and the systems approach, plus the suggestions from the four 
input sessions, the emphasis in the evaluation design was changed from one of looking primarily 
at past accomplishments and current status to significantiy more emphasis on the future. This 
change is graphically depicted in Figure 6. As a result of this change, the framework places 
more emphases on the preparation of mission and vision statements, development of a strategic 
plan, and preparation of an operational plan for the vocational program as a part of the evaluation 
process. Data on the current status of the vocational program will be collected and used. One of 
the major emphasis in the evaluation process will be to develop and validate strategic and 
operational plans that will make the vocational program efficient, effective and productive. 

The framework is described in more detail in the paragraphs that follow. Suggestions from the 
input sessions and the advisory committee were used to develop the framework. 

The three major components in the vocational program evaluation process are: 

• Self-assessment and planning. 

• External validation. 

• Action plan implementation. 
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FIGURE 4: Desired System 
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The purpose of the self-assessment and planning phase is to determine the status of the program, 
identify needs, and evolve a plan. The status of the program will be determined by input from 
various stakeholders in the program, a review of data collected on various processes and outputs 
of the program, and information from the measures and standards established by DPI. With the 
exception of the measures and standards which are mandated by the Carl Perkins Act, the other 
indicators would be selected by people associated with the program. This selection process will 
be guided by the vision and mission statements developed for the program. (See Figure 7) As a 
part of this process, the stakeholders need to determine the boundaries of the program to be 
evaluated. Are the boundaries based on the senior high school vocational courses, middle and 
high school courses, or the total school-to-work transition program? 

Needs are identified by contrasting current performance levels with the desired or required 
performance levels. The larger the discrepancy the larger the need. These discrepancies will be 
reviewed by the local district staff, advisory committee members and other stakeholders to 
determine their overall significance and priority. 

The mission and vision statements and the needs identified form the basis for developing a 
strategic plan for the district's vocational education program. Also, the mission statement will 
define the scope of the program to be evaluated. The linkage between these elements in the 
planning system need to be apparent and logical. The strategic plan will set the overall direction 
for a three- to five-year time period. It will contain goals and strategies for attaining these goals. 

The districts operational plan will evolve from its strategic plan. The operational plan will 
include one year objectives. In some instances the district may want to plan for two years. In 
addition to the objectives there will be activities identified to meet these objectives, 
responsibilities assigned, and evaluation criteria listed. A resource budget needs to be identified 
for the operational plan. This would indicate how human resources, facilities, equipment and 
operational funds will be used to accomplish the plan. 

The self-assessment and planning processes will be managed and conducted by the local school 
district. Working within the TQM framework, it is important that these processes be conducted 
and completed by a team comprised of program stakeholders. Districts should consider going 
beyond their staff members in selecting team members. Representatives from local business and 
industry, parents, students, and graduates should be considered for participation on the team. 
The team will need to identify its role and responsibilities and how it interfaces with the 
administrative structure of the local district. All of the staff members involved in the vocational 
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FIGURE 6: Change In Emphasis 
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education program must be involved in this team. If the team becomes too large, it can be 
divided into several work teams. 

The local district will identify its customers, the types of data collection methods to be used, and 
the areas to be evaluated. The exceptions to this would be the areas that are mandated by the 
Carl Peiicins Act and/or Wisconsin Department of Public Instruction. In addition, other federal 
regulations may be involved. 

The committee and program staff members wiU develop the vision and mission statements, 
strategic plan and the operational plan. A school may want to involve a consultant or consultants 
in this phase. Care should be exercised in selecting a consultant to assure that this individual will 
be able to provide expertise in the areas needed. 

The external validation phase of the evaluation replaces the on-site evaluation phase from the 
former evaluation model. The title for this phase has been changed to reflect the change in 
emphasis in this portion of the evaluation system. In the previous model, a team of evaluators 
reviewed the self-evaluation report, collected data during the on-site visit, and determined 
whether the program had achieved cenain goals and expectations. In the new evaluation model, 
the primary focus of this phase is to review the district's strategic and operational plans for its 
vocational program. An outline of the activities involved in the external validation is included in 
Figure 8. 

Local school districts will have considerable flexibility in selecting the members of the external 
validation team. A team leader will be needed. This person should have experience in the on- 
site validation process and in team leading. The number of "additional team members will depend 
upon the scope of the program and the types of input needed. The validation team could be 
selected from educators, consultants, business/industry representatives, school board members, 
graduates, and/or other groups. For example, a school district may wish to focus a part of the 
validation visit on the school-to-work transition portion of its program. To do this, consideration 
would be given to having counselors, vocational teachers, and a personnel or human resource 
specialist from B/I on the team. If there is an apparent need to make significant changes in one 
of the vocational curriculum areas, it may be desirable to have one or more consultants from that 
area participate on the external validation team. 

The third phase of the evaluation system is action plan implementation. The actions in this phase 
were based on the results of the first two phases. At the end of the external validation team visit. 



ERIC 



17 24 



the local district will have a validated action plan. This action plan will include the strategic 
plan, operational plan, and action plans for the team or teams involved in the implementation 
process. In addition to action plans for the teams there also need to be individual action plans for 
staff members. Each staff member needs to develop his/her own plan. The team plans and 
individual plans need to be aligned with the overall operational plan and strategic plan for the 
program. (See Figure 9) 

When these three phases arc used in a systematic manner, the vocational program will experience 
conHnuous improvement. This is depicted in Figure 10. Self-assessment and planning are based 
on information available to the local staff members and other stakeholders in the program. This 
plan is validated by an external team. This step provides a check on the processes and 
procedures used by the internal team. It also is an opportunity to acquire new insights and 
information fi om the validation team. 

The next step is to put the validated plan mto operation. When the plan is implemented, its 
impacts and outcomes must be evaluated to provide information for the next cycle of the 
evaluation process. One of the principles of systems design is to maximize the effectiveness of 
the system rather than emphasizing a component. This- principle must be kept in mind as the 
plan is developed and implemented. 
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FIGURES: External Validation 
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FIGURE 10: Continuous Improvement Cycle 
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Recommen dations for the Evaluation System 

The results from the four input sessions and the survey were reviewed with the project's advisory 
committee on Friday, April 23, 1993. The researcher also presented the TQM, systems, and 
evaluation concepts presented in the figures reproduced in this paper. The committee members 
agreed with the following recommendations for the evaluation system. A list of committee 
members is given in Appendix B. 



1. Develop an evaluation system that goes beyond the data required in the Standards and 
Measures. ^ 

2. Coordinate the evaluation system with the Standards and Measures and other data 
required by State and federal programs. 

3. Design the evaluation system to help improve local programs. 

4. Design the system to evaluate the School-to- Work Transition Program. 

5. Use a systems approach to evaluation. 

6. Apply TQM concepts in designing the system, 

• Meet customer expectations. 

• Process improvement. 

• Team and individual empowerment. 

• Continuous improvement system. 

7. Change the evaluation system to focus on the future. 

8. Continue to use the three phases in the original evaluation system, but modify them to 
be future oriented and more efficient. 

• Self-assessment and planning- 

• External validation. 

• Implementation of the Action Plan. 
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Appendix A 



Input Session Results 

• Menomonie A-1 

♦ Madison A-2 

• Appleton A-5 

♦ Wausau A-7 

Participants in Input Sessions. A-9 

Sample Survey Form A- 12 



Input From the Session at Menomonie 
November 9, 1992 

The first input session for tlie revision of the Secondary Vocational Education Program^Evaluation 
System was held on the campus of UW-Stout on November 9, 1992, This session was attended 
by eight people. 

The results of the discussion and prioritization processes are given below: 

I . Components of the Current Evaluation System that should be retained are listed in priority 
order below. The number in parentheses after the statements indicate the number of votes 
received in the prioritization process. The larger the point total, the higher the priority 
placed en the recommendation. 

1 . A structured system for the evaluation process should be retained. (33) 

♦ Exit interview. 

♦ Exit report. 

♦ Evaluate against standards-"What Should Be." 

♦ Key questions. 

♦ Training for team leaders and team members. 

♦ Draft report available at the end of the on-site. 

♦ Opportunity for the local LVEC to address the on-site team. 

2. Continue the self-evaluation portion of the process. (29) 

3 . Retain the external evaluation team and the on-site visit. (28) 

4. Include the involvement of guidance and counseling and special needs programs. (16) 

5 . Continue the graduate follow-up process. (10) 

Other recommendations and votes received were: 

Continue the strong relationship with the university. (2) 
♦ Continue the interchange of ideas on vocational educatic i through the opportunity to 

serve on on-site teams. (2) 

II. Changes that should be made in the evaluation process. 

1 . Make more provision for tlie school district to discuss/explain priorities and Ions-range 
plans in the self-evaluation report and to the on-site team. (22) 

2. Expand the preparation for the on-site team and tlie communication with team 
members prior to the time they come on-site. (20) 

3. Provide a better definition of "Needs Assessment." (18) 

A . .Improve the survey forms and data collection processes. ( 1 6) 

♦ SimpHfy the fonns - less questions, narrow the focus. 

♦ Consider omitting student names on the surveys. 

♦ Use optical scanning and oihcr data management tecluiiqiics to improve 
turnaround time. 

5. I:x|)and tlic business/industry folio w-up survey process in include input on what 
competencies they want high school graduates to have, (14) 
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6. Change the process for selecting team members. Obuiin more commitment and district 
support for providing team members. (12) 

7. Develop a system for identifying the "Vocational Education Student." (II) 
Other suggestions that did not receive as much priority were: 

♦ Schools need to develop better processes for utilizing the expertise of evaluation team 
members. 

♦ Cluster LVECs need options that fit their multiple school situation. 

♦ Change the exhibit/display area of documents. Much of the information provided 
during the on-site evaluation was not used. 

III. New components that need to be added to the evaluation system. 

1 . Expand the evaluation to include Tech Prep, Education for Employment and integration 
of other subject matter areas with vocational education. (37) 

2. Change the structure of the on-site evaluation team. (26) 

♦ Add academic teachers. 

♦ Include people from business and industry. 

♦ Include community members. 

♦ Include technical college staff. 

♦ Include university faculty members. 

3 . Include an outcome-based assessment component (13) 

4. Provide more flexibility in the evaluation process. (11) 

♦ Address local goals. 

♦ Link to local plan. 

5 . Add student performance assessment processes. (9) 
Odier suggestions and votes received were: 

♦ Make provision in the evaluation process to address pilot and/or exemplary 
projects/programs at the district level. (6) 

♦ Develop techniques to make more effective use of previous evaluation data and 
recommendations. (6) 

Make tlie evaluation a continuous process. (5) 

♦ Provide the capacity to address local, state and/or national issues. (5) 

♦ Place more emphasis on staff development. (2) 

♦ Provide data on critical resources and practices such as release time for curriculum 
development, supervision of cooperative education students, etc. 
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Input From the Session at Madison 
November 23, 1992 



The group responded to the three basic questions established for the input sessions. 

I. The following are the components from the secondary vocational education evaluation 
system that the participants wanted to keep. NOTE: Tlie participants also suggested some 
changes within these components. These will be noted in Section II. These 
recommendations are not listed in priority order. 

A. Questions on the curriculum areas were useful. They encouraged the evaluators to 
look at various aspects of the curriculum. 

B . The Stout coordinator was important to the process. 

C. It was important to have data processing available. 

D. The overall fomiat for the evaluation should be retained. This includes the self- 
evaluation, on-site team process and utilization activities. 

E. The follow-up of graduates and employers should be retained. 

F. Include student surveys in the new evaluation process. 

G . If possible, keep the data elements parallel to the data elements in the old evaluation 
in order to provide comparable data. In the discussion it was noted that this is not a 
top priority, but is sometliing that should be done if it is possible to accomplish 
without putting significant constraints on the new evaluation design. 

II. Components and elements in the old evaluation process that should be changed and/or 
dropped. These are presented in priority order. The number at the end of the statement 
indicates the total number of points received in the prioritization process. There was a 
maximum of 50 points tliat could be given any one item. 

1 . Improve the follow-up survey process and forms. (34 Points) 

• Shonen the forms to make them more efficient and improve response rates. 

• Acquire more information on the future-what vocational programs should 
emphasize to prepare students for future requirements. 

• Include a variety of ways to complete the surveys-such as telephone interviews, 
focus groups, sampling, mail surveys, etc. 

• The one-year survey should not be conducted until at least a year has passed. It 
may be better to use a tliree-year survey to give a longer timeframe for the 
graduates to provide input on. 

Combine an exit survey of students along with the one- or three-year survey to 
provide more appropriate data. Input should be obtained on all higli school 
graduates not only the vocational program graduates. In addition, ask them 
about their future competency needs as they [)erceive them. 

• The employer survey should include a random sample of employers and 
community members. 

• Assess graduates' and employers' perceptions of the transition from school to 
work. 
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2. 



Make the evaluation process more efficient. (23 Points) 



Reduce the time period required to put the evaluation together. 
Reduce tiie total amount of time required to do the evaluauon. 
Make the evaluation more cost-effective. 

Design the evaluation so that it makes use of and also provides data for the 
standards and measures required under the Carl Perkins Act, and the data needed 
for other evaluation activities such as SEC and the DPI audits. 

3 . Make provisions for outcome-based education in the evaluation system. (20 Points) 

4. Modify the evaluation team composition. (17 Points) 

• Add people from business and industry, technical colleges, and the university 
system. 

• Have a team comprised of two coordinators direct the on-site evaluation process. 
The team of coordinators could be comprised of university and technical college 
faculty members. 

5 . Change the team visit to make it more flexible, less intense, and provide more options 
for team member activities. (17 Points) 

6 . Involve academic departments and curriculums in the evaluation process. (14 Points) 
There is a need to identify the degree to which academics are applied to vocational 
areas and reinforced in vocational courses. 

7 . Develop the evaluation system so that the tech prep program can be evaluated as a 
part of the process. (1 1 Points) 

8 . Link or relate the evaluation process to the benchmarking system which is being 
developed in the state. In other words, incorporate the best practices identified 
(6 Points) 

9 . Review the questions and guides for the on-site evaluators. These were helpful but 
they need to be reviewed and changed to meet current standards and program 
concepts. If possible, shorten the list of questions. (5 Points) 

1 0. Include the concept of authentic assessment. 

III. Additional Evaluation Needs Identified by the Participants. 

1 . Develop an evaluation model that can be used to evaluate the districts* Education for 
Employment Program. (50 Points) 

Business/education partnerships. 
Applied academics/cross teaching. 
Competency-based education. 
Assessment. 

• Developmental guidance program (Careers Development Program) 
Advanced placement program and processes. 
IJnkages-process and product. 

School-io-work transition processes. 
Equity program. 

• Work sludy/co-op program-work site learning. 
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• Apprenticeship program (Youth apprenticeship). 

Education for Employment councilA^ocational advisory committees. 
Special needs program. 

• Staff in-service program. 
Exploration programs. 

• At-risk and special needs programs (Standard M). Equity and special needs apply 
to the total school program and should be in the total school evaluation model. 

2. Develop targets for quality programs. Use these for each area in the evaluation. 
(38 Points) 

3. Link the evaluation to school's vision statement and strategic plan. Also, tie this in 
with the school repon card. (22 Points) 

4. Design a flexible evaluation system so that schools can select modules that fit their 
needs and program status. (21 Points) 

5 . Network the evaluation system to the Carl Perkins Act standards and measures so that 
. the data do not have to be collected a second time. 
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Input from the Session at Fox Valley Technical College-Appleton 

November 30, 1992 



I. The following components of the original vocational education evaluation system were 
recommended for retention in the new system. There may be some changes in specific 
areas within each component. These changes are noted in sections two and three. The 
recommendations are given in priority order. The point total in parentheses after each item 
is tlie total number of points received in the prioritization process. The more points the 
higher the priority. 

1 . Continue to use an advisory committee to provide input to CVTAE on the evaluation 
system. This committee should have membership from LVECs, Business and ^ 
Industry, Bureau of Vocational Education (DPI), Technical College System, and the 
University Systein. (34 Points) 

2. Continue the process for identifying members for the evaluation teams. Maintain a 
pool of educators and consultants who can serve on evaluation teams. Have the LVEC 
make the evaluation team contacts. (30 Points) 

3. Continue to use the self-evaluation. (24 Points) 

4. Provide training sessions for evaluators. This would include those planning the 
evaluation and evaluation team leaders. (22 Points) 

5 . Continue to provide statistical analysis and graphs services. ( 10 Points) 
II. Changes that need to be made in the vocational education evaluation system. 

1 . Develop a more functional self-evaluation report. The report needs to be smaller, 
contain less education jargon, etc. (24 Points) 

2. Allow local school district flexibility in selecting the composition of the evaluation 
team. A local district should be able to determine if it wants to have consultants or 
peer evaluators. (21 Points) 

3 . Users of the evaluation process and system should have a major input into its direction 
(21 Points) 

4. Deyelop a flexible evaluation cycle. Local districts should be able to select a time 
period m which they evaluate tlieir vocational programs based on when evaluations are 
needed. There should be a maximum number of years in the cycle, such as seven to 
ten years. Make this cycle consistent with other evaluation cycles such as the 

SEC (11 Points) 

5 . Provide flexibility to adjusi to curriculum changes. (9 Points) 

6. Have the evaluation team review and evaluate the implemeniaiion plan as well as the 
current status of the program. (7 Points) 

7 Allow the use of focus groui)s, interviews and, other tccliniques to ratlicr 
data. (6 Points) 



ERIC 



35 



8 . Provide a list of alternative techniques and ways to collect data in each of the areas 
evaluated. (6 Points) 

III. Otlier evaluation needs identified. 

1 . Evaluate the school-to-work transition program. The following components are 
included in this program, but the list is necessarily exhaustive. (36 Points) 

• Education for Employment. 

♦ Each of the educational pathways. 
-College Prep 

-Tech Prep 
-Work Prep 

♦ Applied academics. 

* Gateway assessment (WSAS) 

♦ Secondary options. 

♦ Youth apprenticeship. 

* BusinessAndustry-educational partnerships. 

* Technical college credit (dual credit) 
- JTPA 

2. Work with the School Evaluation Consortium. (SEC) (29 Points) 

♦ Examine linkages between disciplines. 

* Assessing the school-to-work transition program components. 

3. Assess linkages betwee-i. curriculum areas. (22 Points) 

* Check restructuring activities and goals. 

♦ Applied academics 

* Work/coordinate evaluation with SEC 

4. Provide Carl Perkins Act funds for evaluation projects that would assist local districts 
with their evaluation planning and activities. (17 Points) 

5. Relate the evaluation process to changes in education, public concerns oolitics etc 
(16 Points) F ' • 

6 . Select a director of the evaluation process who is knowledgeable of vocational 
education and the school-to-work transition program. (13 Points) 

7 . Do not duplicate the data collection required for VEER's. (2 Points) 
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Input from the Session at Northcentra! Technical College- Wausau 

December 1, 1992 

Because of the weather and slippery roads, only one of four people scheduled to attend tlie session 
was able to attend. Dave Carney from Wisconsin Rapids provided the following input. Since 
there was only one person at the session, no prioritization was made of the items. 

I . What do you want to keep from the secondary vocational education program evaluation 
process? 

A. Review of the current vocational program. 

B. Self^study. 

C. Follow-up study of graduates. 

D. On-site evaluation: The external evaluators, process and report. 

II. What should be dropped or changed in the secondary vocational education program 
evaluation process? 

The following changes in the secondary vocational education evaluation process were 
suggested. 

A. More authentic assessment 

• What is being done-compared to the labor-market needs in the community? 

• Focus groups. 

• More involvement of community, parents, business/industry, etc. 

B . Changes in the self-study. 

• More labor market information. 

- Job service office information. 

• More research on what is appropriate for the program. 

- New jobs. 

- New technology. 

- New teaching techniques. 

C. Follow-up study. 

• Drop the one-year follow-up. 

• Use five- and ten-year follow-up studies. 

• Broaden the scope-include all graduates of the high school. This would require that 
the follow-up fonns be changed. 

D. Oii-siic cvaluaiion changes. 

• Use a longer cycle-seven years. 

• Change ic^mi size-use smaller teams if possible. Schedule less inter\Mcws for each 
of the team members. 

• I lave training sessions for new evaluators. 

• Continue iL . training for team leaders and the I.V[K:s planning the local 
evaluations. 
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♦ On-site evaluation reports should be reviewed by each of the evaluators (writers) 
after they return to their home base. ( This provides reflection time.) 

. E. Eliminate unnecessary data. Reduce the overall vohime and eliminate the 
infoniiation not used. 

F . Implementarion-The evaluation team should identify seven to ten major problem areas 
for the local district to focus on. 

G. Share the results of the evaluations with teacher educators in various vocational areas. 

H. Share the results with local technical college staff. 

III. What otlier evaluation needs do you and your school have in relation to vocational 
education and school-to-work transition programs? 

A. Incorporate the tech prep initiative in the process. Also evaluate the school to work 
transition program. 

B . Identify job oppomnities in tlie community. 

♦ With a high school diploma. 

♦ For those who complete apprenticeships. 

♦ With a technical college degree. 

♦ With a college degree. 

C. Assess foundation skills. (SCANS) 

♦ Where are they taught? 

♦ At what levels are students achieving in each of these competency areas? 
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Participants in the Input Sessions 
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Participants-Menomonic Input Session-November 9. 1992 



Wendy Ottestad-Osseo School District 
Cheryl GulUclc^nid-Osseo School District 
Becky Johnson-Elk Mound School District 
Andy Eaton-Hayward School District 
Sherri Torkelson-Eau Claire School District 
Herb Mehne-Menomonie School District 
Jay Wagner-Chippewa Falls School District 
Gerald Munyon-Chippewa Falls School District 

Participants-Madison Input Session-November 23. 1992 

Donna Ochsner-CESA #3 
Nancy Breunig-Sauk Prairie School District 
Ken Sedbrook-Monroc School District 
Walt Gabrysiak-Kettle Moraine School District 
Bill Herron-Kettle Moraine School District 
Cyndy Sandberg-Watertown School District 
Dick Dignan-WCOVE 
^ Judy Gilbert-Muskego School District 
Alben Pitts-Racine School District 
Jim Olstad-Oregon School District 
Chris Kruegcr-Franklin School District 
Scott Pierce-Franklin School Disuict 
Gene Szymaszek-New Beriin School District 
Russ Plagemann-Madison School District 

Participants-Fox Valley- Appleton Input Session-November 30, 1992 

Gari Spagnolctti-Applcton School District 
Cindy Ebcrt-Ripon School District 
Gloria Smith -Manitowcx: School District 
C^ail Framc-Hovvard-Suamico School District 

Rick Nonhrop-Bayport School District and Moward-Suamico School District 
Jim Krucgcr-Ashwaubcnon School District 
Chuck l^nclcrs-Laona School District 
Ted Schmcr-Oslikosh School Disinct 
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Bill Ratzburg-Kenosha School District 

Robert Enghagen-Burcau of Vocational Education, DPI 

Dan Tentchcr-Winneconne School District 

Participant-Northcentral-Wausau Input vSession-Decenibcr 1, 1992 
Dave Carney-Wisconsin Rapids School District 
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Appendix B 
Project Advisory Committee Members 
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Advisory Committee MccJing 
Secondary Vocational Education Program Evaluation Project 

Meeting Date: Friday, April 23 199T 
'nine: 9:00 a.ni.-3:()0 p.m. 

Place: Board Room, Howard Joluisoii East Town-Madison 
190-94 and 151 

AGENDA 

8:45-9:00 Coffee and Conversation 

9-00 Welcome and Oven.^icw of die Agenda 

9: 1 5-9:25 Brief History of the Original Evaluation Project 

9:25-9:45 Review Current Project 

• Puq^ose 

• Process 

• Major Activities 

• Results-Report of Input Session Recommendations 
9:45-10: 15 Survey Results 

10: 15-Noon Evaluation Design 

• Design Framework 

• Major Concepts Used to Create the Design 
- Components of the Evaluation System 

Noon- 1:00 Lunch 

1 :00-2:30 Discuss the Components of the Evaluation System 

• Discuss 

• Refine 

• Expand 

2:30-3:00 Next Steps in the Project 

3:00 Adjouni 
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Project Advisory Committee Members 



Richard Dignan, Executive Secretary Jim Schultz, LVEC 

Wisconsin Council on Vocational Edgerton High School 

Education 200 Elm High Drive 

30 West Mifflin Street, Suite 301 Edgerton, WI 53534 
Madison, WI 53703 



David Doerfert, Consultant 
Agriculture Education and FFA 
Bureau for Vocational Education 
Wisconsin Department of Public 
Instruction 

125 South Webster Street 
Madison, WI 53707 

Terry Downen, Principal 
Verona Area High School 
300 Richard Street 
Verona, WI 53593 

Robert Enghagen, Planning and 
OCR Specialist 

Bureau for Vocational Education 
Wisconsin Department of Public 
Instruction 

125 South Webster Street 
Madison, WI 53707 



Fred Skebba, LVEC 
Rhinelander High School 
665 Coolidge Avenue 
Rhinelander, WI 54501 



Christine Krueger, LVEC 
8222 S. 51st Street 
Franklin, WI 53132 



Richard Marks 

Technology Education Instructor 
Marshfield High School 
1401 Becker Road 
Marshfield, WI 54449 



44 



